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ABSTRACT 

Introduction: The feedback and public reporting of 
PROMs data aims to improve the quality of care 
provided to patients. Existing systematic reviews have 
found it difficult to draw overall conclusions about the 
effectiveness of PROMs feedback. We aim to execute a 
realist synthesis of the evidence to understand by what 
means and in what circumstances the feedback of 
PROMs data leads to the intended service 
improvements. 

Methods and analysis: Realist synthesis involves 
(stage 1) identifying the ideas, assumptions or 
'programme theories' which explain how PROMs 
feedback is supposed to work and in what 
circumstances and then (stage 2) reviewing the 
evidence to determine the extent to which these 
expectations are met in practice. For stage 1, six 
provisional 'functions' of PROMs feedback have been 
identified to structure our review (screening, 
monitoring, patient involvement, demand management, 
quality improvement and patient choice). For each 
function, we will identify the different programme 
theories that underlie these different goals and develop 
a logical map of the respective implementation 
processes. In stage 2, we will identify studies that will 
provide empirical tests of each component of the 
programme theories to evaluate the circumstances in 
which the potential obstacles can be overcome and 
whether and how the unintended consequences of 
PROMs feedback arise. We will synthesise this 
evidence to (1) identify the implementation processes 
which support or constrain the successful collation, 
interpretation and utilisation of PROMs data; (2) 
identify the implementation processes through which 
the unintended consequences of PROMs data arise and 
those where they can be avoided. 
Ethics and dissemination: The study will not 
require NHS ethics approval. We have secured ethical 
approval for the study from the University of Leeds 
(LTSSP-019). We will disseminate the findings of the 
review through a briefing paper and dissemination 
event for National Health Service stakeholders, 
conferences and peer reviewed publications. 



Strengths and limitations of this study 



Our realist synthesis will clearly articulate the differ- 
ent ideas and assumptions underlying how the dif- 
ferent functions of patient reported outcome 
measures (PROMs) feedback are intended to work. 
The synthesis will identify how, why and in what 
circumstances the feedback of PROMs data leads 
to the intended service improvements. 
The synthesis will integrate qualitative and quan- 
titative evidence. 

The synthesis will provide actionable guidance 
for policymakers to improve the implementation 
of PROMs feedback to take account of local 
circumstances. 

The synthesis is very ambitious and the literature 
in this area is large but uneven, with many more 
studies of PROMs feedback at an individual level 
compared to the aggregate level. 
There are time constraints that may result in the 
team focusing on or prioritising some areas of 
the literature over others. 



BACKGROUND 

Policy context and definitions 

PROMs are questionnaires that measure 
patients' perceptions of the impact of a condi- 
tion and its treatment on their health. 1 Many 
of these measures were originally designed for 
use in research to ensure that the patient's per- 
spective was integrated into assessments of the 
effectiveness and cost-effectiveness of care and 
treatment. 2 These perceptions are now taken 
to be a key indicator of the quality of care 
patients receive. Through the introduction of 
the National PROMs programme in England, 
the feedback and public reporting of these 
data aims to improve the quality of care pro- 
vided to NHS patients. 3 4 

Alongside the use of PROMs data at an 
aggregate level, the routine collection and 
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use of PROMs data at the individual patient level has 
also become more widespread but in a less coordinated 
way, with individual clinicians using them on an ad hoc 
basis, often with little guidance. 5-7 At the individual 
level, the intention of PROMs feedback is to enhance 
communication between patients and clinicians, 
improve the detection of patient problems, support clin- 
ical decision-making about treatment through ongoing 
monitoring and to empower patients to become more 
involved in their care. ' There are inherent tensions 
between the different uses of PROMs data that may 
influence how it is collated and interpreted and thus its 
success. There is a significant need for research that 
clarifies the different functions of PROMs feedback and 
delineates more clearly the processes through which 
they are expected to achieve their intended outcomes. 

Existing evidence 

PROMs feedback is a complex intervention and reviewing 
the evidence of its impact is challenging. PROMs feedback 
is unavoidably heterogeneous and varies by PROM used, 
purpose of feedback, patient population, setting, format 
and timing of feedback, recipients of the information and 
level of aggregation of the data. 8 Systematic reviews using 
traditional methodologies have found it difficult to draw 
overall conclusions about the effectiveness of PROMs feed- 
back or to isolate the precise combination of factors that 
make for its success. 10 11 The implementation chain from 
feedback to improvement has many intermediate steps 
and may only be as strong as its weakest link. 12 At individ- 
ual and aggregate levels there are many organisational, 
methodological and logistical challenges to the collation, 
interpretation and then utilisation of PROMs data. 13 For 
example, at an aggregate level, these include reducing the 
risk of selection bias as older, sicker patients are less likely 
to complete PROMs, 14 reducing the variation in recruit- 
ment rates in PROMs data collection across NHS trusts, 15 
ensuring that procedures are in place to adequately adjust 
for casemix, 16 17 collecting the data at the right point in 
the patient's pathway and summarising this information in 
away that is interpretable to different audiences. 18 

Furthermore the success of PROMs feedback is context 
dependent and these contextual differences influence the 
precise mechanisms through which it works and its impact 
on patient care. For example, using PROMs data as an indi- 
cator of service quality for surgical interventions in acute care 
is very different from its use as a quality indicator of general 
practitioners' management of long-term conditions within 
primary care. 19 The impact of surgery on disease-specific 
PROMs and knowledge of the natural variability of scores has 
been well documented 20 but this knowledge is lacking 
regarding the impact of primary care on EQ-5D scores. 

Differences in context can also result in the intervention 
not working through the intended mechanisms, leading 
to unintended consequences. 21 For example, the feedback 
and public release of performance data may lead to sur- 
geons refusing to treat the sickest patients to avoid poor 
outcomes and lower publicly reported ratings. 22 Data from 



the national PROMs programme has been misinterpreted 
by some as indicating that a significant proportion of vari- 
cose vein, hernia and hip and knee replacement should 
not take place. 23 Public reporting of performance data 
may not improve patient care, as intended, through 
informing patient choice. 24 Rather, patients are often 
ambivalent about performance data and rely on their CP's 
opinion when choosing a hospital. 25 Finally, research 
coverage of PROMs feedback is uneven, with more studies 
(trials and qualitative case studies) examining PROMs 
feedback at an individual level and few studies examining 
their use as a performance indicator at a group level. 26 

In summary, we have identified a number of gaps in 
our knowledge that this review seeks to address: 

► How can the tensions between the different purposes 
of PROMs data collection and their use by different 
stakeholders be resolved? 

► Through what processes or mechanisms are the dif- 
ferent applications of PROMs feedback supposed to 
work? 

► For each application of PROMs data, what are the 
potential obstacles or unintended consequences of 
PROMs feedback that may prevent, limit of constrain 
the intervention improving patient care? 

► For each application of PROMs data, what are the 
implementation processes that enable, facilitate or 
support PROMs feedback to improve patient care? 

Aims and objectives 

As the applications of PROMs data continue to multiply, 
our first aim is to identify and classify the various ambi- 
tions of PROMs feedback. Our objectives are to: 

A. Produce a comprehensive taxonomy of the 'pro- 
gramme theories' underlying these different func- 
tions, capture their subtle differences and the 
tensions that may lie between them and; 

B. Produce logical models of the organisational logis- 
tics, social processes and decision-making sequences 
that underlie the collation, interpretation and utilisa- 
tion of PROMs data. 

We will use these models to identify the potential 
blockages and unintended consequences of PROMs 
feedback which may prevent the intervention achieving 
its intended outcome of improving patient care. This 
will provide a framework for the review. 

To inform the future implementation of PROMs feed- 
back, our second aim is to test and refine these pro- 
gramme theories about how PROMs feedback is 
supposed to work against existing evidence of how it 
works in practice. The specific objectives of this synthesis 
are to: 

A. Identify the implementation processes that support 
or constrain the successful collation, interpretation 
and utilisation of PROMs data and; 

B. Identify the mechanisms and circumstances through 
which the unintended consequences of PROMs data 
arise and those where they can be avoided. 
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Our third aim is to use the findings from this synthesis 
to identify what support is needed to optimise the 
impact of PROMs feedback and distinguish the circum- 
stances or contexts (eg, settings, patient populations, 
nature and format of feedback) where PROMs feedback 
might work best. We will produce guidance to enable 
NHS decisionmakers to tailor the collection and utilisa- 
tion of PROMs data to local circumstances and maximise 
its impact on the quality of patient care. 

Review methodology 

We have chosen to use realist synthesis (RS) as the meth- 
odology for this review because it is specifically designed 
to manage the uneven body of evidence such as is avail- 
able on PROMs feedback. RS is designed to disentangle 
the heterogeneity and complexity of the intervention 
and to make sense of the various contingencies, 
blockages and unintended consequences that may influ- 
ence its success. The methodology was developed by one 
of the coauthors (RP). 27 28 Explanation building 
through theory development, testing and refinement is 
an iterative process that occurs throughout the review. 
This can be broken down into six-stages as outiined in 
figure 1. 

Identifying the review question 

The basic unit of analysis in RS is not the intervention 
but the ideas and assumptions or programme theories that 
underpin it. Thus, the starting point of RS is to cata- 
logue and build logical models of the different ideas 
and assumptions about how interventions are supposed 
to work. In developing the protocol, we have already 
begun to engage with this process and have identified 
six functions of PROMs data that will provide an initial 
structure for the review: At the individual level, PROMs 
data are utilised to improve patient care by (1) screening 
for undetected problems; (2) monitoring patients' pro- 
blems over time and (3) activating patients and involving 
the patient in decisions about their care. At the group 
level, PROMs data may improve patient care by (4) 
improving the appropriateness of the use of interven- 
tions; stimulating quality improvement activities through 
(5) benchmarking provider performance or (6) through 
informing decision-making about choice of provider. 
These are provisional and will be extended and refined 
during the first phase of the review. 

Search strategies: theory 

Searching in RS occurs in two main stages: searches to 
identify the theories underlying the intervention and 
then searches to identify empirical evidence to test these 
theories. We will provide detailed elaboration of the dif- 
ferent programme theories that underlie these different 
functions of PROMs data through (1) electronic search- 
ing and (2) discussion with the NHS stakeholders and 
topic experts on our team and a patient group. All 
search strategies will be tested and refined further 
during the review. 



To identify theories underlying PROMs feedback, we 
will conduct electronic searches of the grey literature to 
identify guidance documentation and policy documents 
and electronic searches of the peer reviewed literature 
to identify position pieces, comments, letters, editorials 
and critical pieces which explain how the different appli- 
cations of PROMs feedback are intended to work. The 
search strategy for electronic databases will comprise 
three search concepts A (PROMs) , B (improvement and 
impact resulting from PROMs feedback including the 
six recognised theories) and C (publication types valu- 
able in eliciting theories). Table 1 lists a small number 
of potential search terms to illustrate the structure of the 
theories search. Further terms and synonyms will be 
tested and added to each concept before the final 
search combination (A AND B AND C) is run to retrieve 
papers discussing theories underpinning the impact of 
PROMs feedback. 

Further searches of Google will explore citations of 
key references found in the electronic database searches, 
and grey literature. We will also conduct electronic 
manual searches of Pulse, the Health Service Journal 
and the BMJ. 

Prioritising theories for review 

To express each programme theory, we will develop a 
logical model of how the mobilisation of PROMs data is 
intended to improve patient care for each function of 
PROMs data. For each application, we will identify the 
potential blockages and unintended consequences of 
PROMs feedback which may prevent or limit the 
achievement of its intended outcome of improving 
patient care. 

For example, one programme theory hypothesises that 
the use of PROMs as an indicator of hospital quality will 
improve patient care by refocusing provider organisa- 
tional priorities on areas of poor performance and initi- 

• 27 

ating quality improvement activities. However, this 
requires a number of organisational logistics and social 
processes to be in place to be successful. The data must 
be collected and collated in a valid way, providers must 
be able to interpret the data and must accept the data as 
a valid reflection of the quality of care. 13 In order to ini- 
tiate quality improvement activities, organisations and 
the individuals within them require the capability and 
resources to control, improve and design processes for 
performance to improve. Furthermore, for change to 
occur organisations require several resources — a reliable 
flow of useful information; education and training in 
the techniques of process improvement; investment in 
the time and change management required to alter core 
work processes; alignment of organisational incentives 
with care improvement objectives; leadership to inspire 
and model care improvement. 28 

It is likely that we will unearth many different ideas 
about how each function of PROMs data is supposed to 
work, their potential blockages and unintended conse- 
quences. It will not be possible to review them all thus 
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Figure 1 Flow diagram for 
realist synthesis. 
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PROMS functionality and feedback: Realist synthesis timeplan. 

1 Identifying the review question 

2 Searching for primary studies 
3. Quality Appraisal 

4 Extracting the data 

5 Synthesising the findings 

6 Dissemination 



Feedback and iteration 
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we will prioritise which programme theories to review in 
two workshops with NHS decisionmakers, including clin- 
icians, NHS managers, policymakers and patients. 

Search strategies: evidence/reviews and primary studies 

The programme theories or hypotheses will provide the 
backbone of the review and determine the search strat- 
egy and decisions about study inclusion into the review 
in order to test and refine these theories. The next stage 
of the review thus involves an evidence search to identify 
primary studies that will provide empirical tests of each 
component of the theory. A starting point will be exist- 
ing quantitative and qualitative systematic reviews of 
PROMs feedback, 29-31 performance reporting 11 and 
clinician and patient interpretation of performance 
data. 32 The results of these reviews will provide valuable 
information on outcome patterns (quantitative 



Table 1 

terms 



Theory search concepts and example search 



Search Concepts 



A— PROMs 



B— PROMs C— Publication 
related types valuable 

Impact or in eliciting 
Improvement 'theories' 



Example 

index 

terms 



Example 
text word 
terms 



Outcome 

Assessment 

(Health Care)/ 

Patient 

Satisfaction/ 

Public* 

reported 

measure* 

Patient* 

outcome* 

measure* 



Feedback/ 

Quality 

Improvement/ 



Comment/Letter/ 
Editorial/ 



Improve* 

Impact* 

Implement* 



Comment on 

Opinion* 

View* 



'indicates truncation. 



systematic reviews) and potential mechanisms (qualita- 
tive systematic reviews). However, we also anticipate the 
need to examine specific individual studies within the 
review to test hypotheses about contextual differences in 
the ways in which PROMs feedback works. Additional 
studies will be identified through searching electronic 
databases. We will search the Cochrane Library, Medline, 
Embase and the researcher's own personal libraries for 
existing reviews with a focussed PROMs search (a com- 
bination of concepts A AND B AND C from table 2. 
Scoping searches indicate 100-200 review abstracts 
would be retrieved. Initial searching indicates that 
reviews alone will not provide current evidence that 
covers known (and unknown) theories. Where relevant 
reviews are outdated or do not adequately cover our the- 
ories we will search electronic databases for relevant 
studies. 

Databases for the evidence search will include 
MEDLINE, EMBASE, HMIC (all Ovid), Health Business 
Elite (NHS HDAS) PsycINFO (Ovid), Science Citation 
Index, Science Conference Proceedings Citation Index 
(Thomson Reuters) and Dissertation and Theses 
(ProQuest). All searches will consist of two concepts 
from table 2 combined as follows (A AND D) but some 
may include a third concept (E in table 2) to limit the 
results to particular empirical evidence publication 
types or study designs (search combination A AND D 
AND E). Searches will include subject headings and text 
words. The concept D searches will be constructed 
individually and will each relate to a function and/or 
impact of PROMs identified at various points along the 
logical models developed in phase 1. For example, one 
concept D search will comprise terms for the function 
of PROMs feedback as a method of screening for 
undetected feedback using subject headings and text 
words such as; 'mass screening'/, 'undetected problem', 
'undiscovered condition'. Initial scoping searches indi- 
cate each theory search will identify 200-800 references 
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depending on the extent of evidence for the theory 
(concept 3), however, since the theories search and 
reviews search will inform and determine these further 
evidence searches it is difficult to estimate at this stage. 
We anticipate developing at least six 'evidence' searches 
to search for evidence on each theory identified in 
phase 1. Further relevant material will be sought from 
targeted searches of websites for example, PROQOLID, 
Google. We will also make extensive use of citation and 
reference tracking to identify relevant papers. 33 

Data extraction and quality appraisal 

These are combined in RS. Quality appraisal will be con- 
ducted throughout the review process and go beyond 
the traditional approach that only focuses on the meth- 
odological quality of studies. 34 In RS, assessment of study 
rigour occurs alongside an assessment of the relevance 
of the study and occurs throughout the process of syn- 
thesis. Quality appraisal is performed on a case-by-case 
basis, as appropriate to the method utilised in the ori- 
ginal study. Where appropriate, we will use relevant 
methodological checklists (eg, GASP) to assess the meth- 
odological quality of included studies. We will also make 
early use of the standards and guidelines emerging from 
the Realist And MEta-narrative Evidence Syntheses: 
Evolving Standards (RAMESES) project. 35 

Different fragments of evidence are thus sought and 
utilised from each study. Each fragment of evidence 
needs to be appraised, as it is extracted, for its relevance 
to theory testing and the rigour with which it has been 
produced. 34 Data extraction requires active engagement 
with each document through note taking and text anno- 
tation. Evidence will be compiled, stored and annotated 
using a series of working papers. 

Owing to the large volume of research in this area, we 
anticipate the need to divide up the process of theory 
testing and thus selecting, abstracting and synthesising 
the data among the review team. Subgroups will be 
formed; one person will lead the process and will share 
and discuss the emerging synthesis with subgroup 
members who will also read a subsample of the papers 
reviewed. 36 The whole project team will also review the 
emerging synthesis through a series of working papers 
shared among the group that will be discussed via face 
to face and virtual meetings. 

Synthesis of evidence 

The goal of realist synthesis is to refine our understand- 
ing of how the programme works and the conditions 
and caveats that influence its success, rather than offer- 
ing a verdict, descriptive summary or mean effect calcu- 
lation on a family of programmes. Specifically, our 
synthesis is concerned with understanding the condi- 
tions in which blockages or unintended consequences 
occur (which may prevent or limit the impact of PROMs 
on patient care) and those in which these blockages can 
be overcome. Synthesis takes several forms. At its most 
basic realist synthesis is a form of 'triangulation', 



bringing together information from different primary 
studies and different study types to explain why a 
pattern of outcomes may occur. For example, systematic 
reviews of PROMs feedback at the individual level dem- 
onstrate a pattern of outcomes such that PROMs feed- 
back influences communication within the consultation 
and increases the detection of problems but has much 

29 37 

less impact on patient management or outcomes. 
This pattern of outcomes suggests there is a 'blockage' 
or 'obstacle' in the implementation chain between dis- 
cussing PROMs findings in the consultation and subse- 
quent action on these findings by clinicians. We can 
then explore potential explanations for the mechanism 
through which this blockage occurs and the circum- 
stances in which it might arise. 

For example, one longitudinal study exploring 
changes in the content of the discussion in the consult- 
ation over time following PROMs feedback found that 
PROMs feedback increases the number of times that 
clinicians discuss symptoms with patients, but not psy- 
chosocial functional issues. 38 Thus, if functional issues 
are not discussed, they are unlikely to be addressed; but 
why are they not discussed? We can look to further quali- 
tative studies for explanation. A focus group study of 
clinicians' and patients' views of the utility of PROMs in 
clinical practice found that clinicians raised concerns 
that PROMs may raise issues that they cannot do any- 
thing about. 39 This suggests that perhaps one reason 
issues are not discussed, despite the availability of 
PROMs data, is that clinicians perceive they cannot do 
anything about the problem. In line with this explan- 
ation, a qualitative study of oncology consultations in 
which PROMs data were available found that doctors 
closed down discussions or did not offer treatment if the 
problem identified was not perceived as a problem for 
the patient or perceived as outside of the remit of the 
clinician. 40 41 Further support for this explanation 
comes from another qualitative study examining com- 
munication within oncology consultations which found 
that oncologists used a number of different strategies to 
close down the discussion of problems they perceived as 
outside of their remit to address. 41 This 'mini-synthesis' 
illustrates how different studies can be brought together 
to understand progress (or otherwise) along the imple- 
mentation chain and identify the circumstances in 
which blockages occur or may be overcome. 

Another form of synthesis, particularly useful when 
there is disagreement on the merits of an intervention is 
to 'adjudicate' between the contending positions. This is 
not a matter of providing evidence to declare a certain 
standpoint correct and another one invalid. Rather adju- 
dication assists in understanding the respects in which a 
particular programme theory holds and those where it 
does not. For example, some studies show that poor per- 
forming hospitals are more likely to engage in quality 
improvement activities following performance feed- 
back, 42 suggesting the feedback works through motivat- 
ing hospitals to improve care. However, other studies 
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Table 2 Evidence search concepts and example search terms 



Search Concepts 



A— PROMs 



B— PROMs related 
Impact or 
Improvement 



C — Review 
publication type 



D— Specific PROMS 
Feedback function or 
impact for example, 
Screening 



E — Empirical 
evidence 

publication types or 
study designs 



Example 
Index terms 



Example 
text word 
terms 



Outcome 
Assessment 
(Health Care)/ 
Patient Satisfaction 
Public* reported 
measure* 
Patient* outcome* 
measure* 



Feedback/quality 

improvement/ 

decision-making 

Improve* 

Impact* 

Implement* 



Use 'reviews 
(maximizes 
specificity)' limit 

Review* near to 
outcome* or 
data* or PRO 



Diagnostic errors/ 
Incidental findings/ 



Screen* 

Undetected problem* 
Undiscover* condition* 



Observational Study/ 
Randomized 
Controlled Trial/ 



Evaluation study 

Follow-up 

Survey* 



have found that poor performing hospitals may experi- 
ence lowered morale and respond by focusing on what 
is measured to the exclusion of other aspects of care. 43 
Our review aims to identify explanations for these con- 
trasting findings to identify the circumstances in which 
the intended mechanism (motivation to improve) and 
intended outcomes occur (improved care) and those in 
which unintended mechanisms (lowered morale, tunnel 
vision) occur. Thus, it seeks to provide an explanation 
for the whole pattern of outcomes across studies rather 
than seek out an average effect. 

Finally, the main form of synthesis is known as 'contin- 
gency building'. All PROMs feedback programmes make 
assumptions that they will work under implementation 
conditions A, B, C and applied in contexts P, Q, R. The 
purpose of the review is to refine many such hypotheses, 
enabling us to say that, more probably, A, C, D, E and P, 
Q, S are the vital ingredients. For example, there is 
debate about whether the public reporting of hospital 
performance data is a necessary ingredient to stimulate 
quality improvement activities, 42 or whether such 
improvement would occur with private feedback to hos- 
pitals alone. 44 Others have argued that the public 
reporting and pay for performance can result in greater 
improvements than public reporting alone. 45 Our review 
would seek to identify the necessary conditions under 
which the feedback of performance data results in 
improvements to the quality of care. Similarly, there is 
also a hypothesis that the feedback of PROMs data at an 
individual level is more likely to improve patient out- 
comes if it is accompanied by management guide- 
lines. 46 47 Our review will test and refine these 
hypotheses against the empirical literature to identify 
the conditions, or contingencies that optimise the 
impact of PROMs feedback on patient care. 

We acknowledge that this is a complex and ambitious 
review. It may not be possible to produce a comprehen- 
sive review of all six of the different functions of PROMs 
we have so far identified. However, we will ensure that 
our review prioritises issues of importance to NHS deci- 
sionmakers through their involvement at key stages in 
the review. 



ETHICS AND DISSEMINATION 

The study will not require NHS ethics approval. We have 
secured ethical approval for the study from the University 
of Leeds (LTSSP-019). The protocol has been registered 
on the PROSPERO database (CRD42013005938). The 
findings from our review will be used to produce a 
number of different outputs and disseminated through a 
number of different mechanisms including a briefing 
document containing guidance to NHS managers, pre- 
sentations at national and international conferences 
including the Annual National PROMs Summit (for 
NHS staff), the Annual PROMs Research Conference 
(held by the London School of Hygiene and Tropical 
Medicine and the King's Fund) and the International 
Society for Quality of Life Research (ISOQOL) confer- 
ence, peer reviewed publications in academic journals 
(eg, BMJ) and articles in practitioner and management 
journals and newsletters (eg, the Health Services Journal, 
Pulse) . 
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